diff --git a/swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/README.md b/swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/README.md index 48aea1fb..e4a39629 100644 --- a/swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/README.md +++ b/swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/README.md @@ -1,163 +1,163 @@ # generate-weblabels-webpack-plugin This plugin for webpack >= 4 enables to generate an HTML page stating the licenses of every source files bundled in the JavaScript assets generated by webpack. Such a page is called [Web Labels](https://www.gnu.org/licenses/javascript-labels.html) and is intended to be consumed by the [GNU LibreJS](https://www.gnu.org/software/librejs/) Firefox plugin whose purpose is, quoting its documentation: > GNU LibreJS aims to address the JavaScript problem described in Richard Stallman's article [The JavaScript Trap](http://www.gnu.org/philosophy/javascript-trap.html). LibreJS is a free add-on for GNU IceCat and other Mozilla-based browsers. It blocks nonfree nontrivial JavaScript while allowing JavaScript that is free and/or trivial. So, without the Web Labels page, the loading of JavaScript assets generated by webpack in client browsers will be blocked by the LibreJS Firefox plugin. Usually, most of the bundled JavaScript source files are retrieved through `npm` or `yarn` and their associated licenses are compatible with those allowed by LibreJS. If your webpack project is also released under a free license, you can use this plugin to ensure LibreJS will detect the licenses of your bundled source files and allow the loading of the webpack generated Javascript assets. In order to be compliant with the LibreJS specifications, the Web Labels page must contain the following information. For each JavaScript asset generated by webpack, all bundled source files in it need to be referenced along with their licenses but also a link to their non-minified source code. The plugin works by processing the compilation statistics available after the whole webpack processing and currently does the following: - For each bundled JavaScript source files, it will try to find its associated LibreJS compatible license by parsing the corresponding [SPDX License Expression](https://spdx.org/licenses/) (usually located in `package.json` files)
- It copies all non-minified source files, bundled into the generated JavaScript assets, into a directory located into the webpack output folder. Also if a license file can be found for a source file it will also be copied.
- It generates either a Web Labels HTML page named `jslicenses.html` or a JSON file named `jslicenses.json`, into a directory located into the webpack output folder. The JSON file should then be used with an HTML template engine to generate the Web Labels page. Its structure is the following: ```json { "": [ { "id": "", "licenses": [ { "name": "", "url": "", "copy_url": "" }, ... ] "src_url": "" }, { ... } ], "": [ ... ], ... } ``` In order for LibreJS to consume the Web Labels, a link to it must be added in the relevant HTML pages of your web application, more details in the relevant [section](https://www.gnu.org/software/librejs/free-your-javascript.html#step3) of the LibreJS documentation. ## Usage To begin, you will need to install `generate-weblabels-webpack-plugin` (**WARNING: wip, this is currently not published on the npm registry**) ```shell $ npm install generate-weblabels-webpack-plugin --save-dev ``` or ```shell $ yarn add generate-weblabels-webpack-plugin --dev ``` Then add the plugin to your webpack configuration, for example: ```js const GenerateWebLabelsPlugin = require('generate-weblabels-webpack-plugin'); module.exports = { // ... plugins: [ new GenerateWebLabelsPlugin(options) ] }; ``` Also, when compiling your assets in production mode, the following webpack option must be adjusted: ```js module.exports = { // ... optimization: { concatenateModules: false } }; ``` This will prevent webpack to concatenate some modules into a single one, which makes license and original source files detection impossible. Please also note that the plugin will not remove the previously copied Web Labels related files into the webpack output folder. You should use the [`clean-webpack-plugin`](https://github.com/johnagan/clean-webpack-plugin) to perform that task. ## Options The following options can be provided ### `outputDir` type: `String` default: `'jssources'` Specify the directory inside the webpack output folder in which the Web Labels related files will be emitted. ### `outputType` type: `Enum` default: `'html'` Specify which type of output the plugin generates. Possible values are: - `'html'`: generate an HTML page named "jslicenses.html" containing the Web Labels - `'json'`: generate a JSON file named "jslicenses.json" containing all relevant info to generate the Web Labels page through an HTML template engine. ### `exclude` type: `Array` default: `[]` An array of module name or source file path prefixes to exclude from the plugin processing. If the prefix does not start with `./`, it is considered as a node module. This can be used to exclude some special modules processed by webpack that do not generate any JavaScript that will be bundled in the generated assets (for instance the scripts associated to [`mini-css-extract-plugin`](https://github.com/webpack-contrib/mini-css-extract-plugin)). ### `srcReplace` type: `Object` default: `{}` An object mapping source files to replace in the generated output. This should be used when a node module points to its minified version by default. ### `licenseOverride` type: `Object` default: `{}` An object mapping source files or node modules to their corresponding SPDX license expressions. This should be used when you have source files inside your webpack project source tree with licenses different from the one of your project. It can also be used when you have a node module with an invalid SPDX license expression in its `package.json`. This object must have the following structure: ```json { "": { "spdxLicenseExpression": "", "licenseFilePath": "" }, ... } ``` The keys of this object can correspond either to a source file path relative to the webpack root folder or a node module identifier. The path of the license file must be relative to the webpack project root folder. ### `additionalScripts` type: `Object` default: `{}` An object declaring additional JavaScript assets loaded by your web application but not generated by webpack. It must have the following structure: ```json { "": [ { "id": "", "path": "", "spdxLicenseExpression": "", "licenseFilePath": "" }, { ... } ], ... } ``` -The paths of the license and source files must be relative to the webpack project root folder. \ No newline at end of file +If the path for a Javascript asset or a license file does not correspond to an url or does not start with '/', it is considered to be relative to the webpack project root folder. \ No newline at end of file diff --git a/swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/index.js b/swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/index.js index 61b22a95..86928ba9 100644 --- a/swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/index.js +++ b/swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/index.js @@ -1,381 +1,382 @@ /** * Copyright (C) 2019 The Software Heritage developers * See the AUTHORS file at the top-level directory of this distribution * License: GNU Affero General Public License version 3, or any later version * See top-level LICENSE file for more information */ // This is a plugin for webpack >= 4 enabling to generate a Web Labels page intended // to be consume by the LibreJS Firefox plugin (https://www.gnu.org/software/librejs/). // See README.md for its complete documentation. const ejs = require('ejs'); const fs = require('fs'); const log = require('webpack-log'); const path = require('path'); const schema = require('./plugin-options-schema.json'); const spdxParse = require('spdx-expression-parse'); const spdxLicensesMapping = require('./spdx-licenses-mapping'); const validateOptions = require('schema-utils'); const pluginName = 'GenerateWebLabelsPlugin'; class GenerateWebLabelsPlugin { constructor(opts) { // check that provided options match JSON schema validateOptions(schema, opts, pluginName); this.options = opts || {}; this.weblabelsDirName = this.options['outputDir'] || 'jssources'; this.outputType = this.options['outputType'] || 'html'; // source file extension handled by webpack and compiled to js this.srcExts = ['js', 'ts', 'coffee', 'lua']; this.srcExtsRegexp = new RegExp('^.*.(' + this.srcExts.join('|') + ')$'); this.chunkNameToJsAsset = {}; this.chunkJsAssetToSrcFiles = {}; this.packageJsonCache = {}; this.packageLicenseFile = {}; this.exclude = []; this.copiedFiles = new Set(); this.logger = log({name: pluginName}); // populate module prefix patterns to exclude if (Array.isArray(this.options['exclude'])) { this.options['exclude'].forEach(toExclude => { if (!toExclude.startsWith('.')) { this.exclude.push('./' + path.join('node_modules', toExclude)); } else { this.exclude.push(toExclude); } }); } } apply(compiler) { compiler.hooks.done.tap(pluginName, statsObj => { // get the stats object in JSON format let stats = statsObj.toJson(); this.stats = stats; // set output folder this.weblabelsOutputDir = path.join(stats.outputPath, this.weblabelsDirName); this.recursiveMkdir(this.weblabelsOutputDir); // map each generated webpack chunk to its js asset Object.keys(stats.assetsByChunkName).forEach((chunkName, i) => { if (Array.isArray(stats.assetsByChunkName[chunkName])) { for (let asset of stats.assetsByChunkName[chunkName]) { if (asset.endsWith('.js')) { this.chunkNameToJsAsset[chunkName] = asset; this.chunkNameToJsAsset[i] = asset; break; } } } else if (stats.assetsByChunkName[chunkName].endsWith('.js')) { this.chunkNameToJsAsset[chunkName] = stats.assetsByChunkName[chunkName]; this.chunkNameToJsAsset[i] = stats.assetsByChunkName[chunkName]; } }); // iterate on all bundled webpack modules stats.modules.forEach(mod => { let srcFilePath = mod.name; // do not process non js related modules if (!this.srcExtsRegexp.test(srcFilePath)) { return; } // do not process modules unrelated to a source file if (!srcFilePath.startsWith('./')) { return; } // do not process modules in the exclusion list for (let toExclude of this.exclude) { if (srcFilePath.startsWith(toExclude)) { return; } } // remove webpack loader call if any let loaderEndPos = srcFilePath.indexOf('!'); if (loaderEndPos !== -1) { srcFilePath = srcFilePath.slice(loaderEndPos + 1); } // iterate on all chunks containing the module mod.chunks.forEach(chunk => { let chunkJsAsset = stats.publicPath + this.chunkNameToJsAsset[chunk]; // init the chunk to source files mapping if needed if (!this.chunkJsAssetToSrcFiles.hasOwnProperty(chunkJsAsset)) { this.chunkJsAssetToSrcFiles[chunkJsAsset] = []; } // check if the source file needs to be replaces if (this.options['srcReplace'] && this.options['srcReplace'].hasOwnProperty(srcFilePath)) { srcFilePath = this.options['srcReplace'][srcFilePath]; } // init source file metadata let srcFileData = {'id': this.cleanupPath(srcFilePath)}; // find and parse the corresponding package.json file let packageJsonPath; let nodeModule = srcFilePath.startsWith('./node_modules/'); if (nodeModule) { packageJsonPath = this.findPackageJsonPath(srcFilePath); } else { packageJsonPath = './package.json'; } let packageJson = this.parsePackageJson(packageJsonPath); // extract license information, overriding it if needed let licenseOverridden = false; let licenseFilePath; if (this.options['licenseOverride']) { for (let srcFilePrefixKey of Object.keys(this.options['licenseOverride'])) { let srcFilePrefix = srcFilePrefixKey; if (!srcFilePrefixKey.startsWith('.')) { srcFilePrefix = './' + path.join('node_modules', srcFilePrefixKey); } if (srcFilePath.startsWith(srcFilePrefix)) { let spdxLicenseExpression = this.options['licenseOverride'][srcFilePrefixKey]['spdxLicenseExpression']; licenseFilePath = this.options['licenseOverride'][srcFilePrefixKey]['licenseFilePath']; let parsedSpdxLicenses = this.parseSpdxLicenseExpression(spdxLicenseExpression, `file ${srcFilePath}`); srcFileData['licenses'] = this.spdxToWebLabelsLicenses(parsedSpdxLicenses); licenseOverridden = true; break; } } } if (!licenseOverridden) { srcFileData['licenses'] = this.extractLicenseInformation(packageJson); let licenseDir = path.join(...packageJsonPath.split('/').slice(0, -1)); licenseFilePath = this.findLicenseFile(licenseDir); } // copy original license file and get its url let licenseCopyUrl = this.copyLicenseFile(licenseFilePath); srcFileData['licenses'].forEach(license => { license['copy_url'] = licenseCopyUrl; }); // generate url for downloading non-minified source code srcFileData['src_url'] = stats.publicPath + path.join(this.weblabelsDirName, srcFileData['id']); // add source file metadata to the webpack chunk this.chunkJsAssetToSrcFiles[chunkJsAsset].push(srcFileData); // copy non-minified source to output folder this.copyFileToOutputPath(srcFilePath); }); }); // process additional scripts if needed if (this.options['additionalScripts']) { for (let script of Object.keys(this.options['additionalScripts'])) { let scriptFilesData = this.options['additionalScripts'][script]; - if (script.indexOf('://') === -1) { + if (script.indexOf('://') === -1 && !script.startsWith('/')) { script = stats.publicPath + script; } this.chunkJsAssetToSrcFiles[script] = []; for (let scriptSrc of scriptFilesData) { let scriptSrcData = {'id': scriptSrc['id']}; let licenceFilePath = scriptSrc['licenseFilePath']; let parsedSpdxLicenses = this.parseSpdxLicenseExpression(scriptSrc['spdxLicenseExpression'], `file ${scriptSrc['path']}`); scriptSrcData['licenses'] = this.spdxToWebLabelsLicenses(parsedSpdxLicenses); let licenseCopyUrl = this.copyLicenseFile(licenceFilePath); scriptSrcData['licenses'].forEach(license => { license['copy_url'] = licenseCopyUrl; }); - if (scriptSrc['path'].indexOf('://') === -1) { + if (scriptSrc['path'].indexOf('://') === -1 && !scriptSrc['path'].startsWith('/')) { scriptSrcData['src_url'] = stats.publicPath + path.join(this.weblabelsDirName, scriptSrc['id']); } else { scriptSrcData['src_url'] = scriptSrc['path']; } this.chunkJsAssetToSrcFiles[script].push(scriptSrcData); this.copyFileToOutputPath(scriptSrc['path']); } } } if (this.outputType === 'json') { // generate the jslicenses.json file let weblabelsData = JSON.stringify(this.chunkJsAssetToSrcFiles); let weblabelsJsonFile = path.join(this.weblabelsOutputDir, 'jslicenses.json'); fs.writeFileSync(weblabelsJsonFile, weblabelsData); } else { // generate the jslicenses.html file let weblabelsPageFile = path.join(this.weblabelsOutputDir, 'jslicenses.html'); ejs.renderFile(path.join(__dirname, 'jslicenses.ejs'), {'jslicenses_data': this.chunkJsAssetToSrcFiles}, {'rmWhitespace': true}, (e, str) => { fs.writeFileSync(weblabelsPageFile, str); }); } }); } cleanupPath(moduleFilePath) { return moduleFilePath.replace(/^[./]*node_modules\//, '').replace(/^.\//, ''); } findPackageJsonPath(srcFilePath) { let pathSplit = srcFilePath.split('/'); let packageJsonPath; for (let i = 3; i < pathSplit.length; ++i) { packageJsonPath = path.join(...pathSplit.slice(0, i), 'package.json'); if (fs.existsSync(packageJsonPath)) { break; } } return packageJsonPath; } findLicenseFile(packageJsonDir) { if (!this.packageLicenseFile.hasOwnProperty(packageJsonDir)) { let foundLicenseFile; fs.readdirSync(packageJsonDir).forEach(file => { if (foundLicenseFile) { return; } if (file.toLowerCase().startsWith('license')) { foundLicenseFile = path.join(packageJsonDir, file); } }); this.packageLicenseFile[packageJsonDir] = foundLicenseFile; } return this.packageLicenseFile[packageJsonDir]; } copyLicenseFile(licenseFilePath) { let licenseCopyPath = ''; if (licenseFilePath && fs.existsSync(licenseFilePath)) { let ext = ''; // add a .txt extension in order to serve license file with text/plain // content type to client browsers if (licenseFilePath.toLowerCase().indexOf('license.') === -1) { ext = '.txt'; } this.copyFileToOutputPath(licenseFilePath, ext); licenseFilePath = this.cleanupPath(licenseFilePath); licenseCopyPath = this.stats.publicPath + path.join(this.weblabelsDirName, licenseFilePath + ext); } return licenseCopyPath; } parsePackageJson(packageJsonPath) { if (!this.packageJsonCache.hasOwnProperty(packageJsonPath)) { let packageJsonStr = fs.readFileSync(packageJsonPath).toString('utf8'); this.packageJsonCache[packageJsonPath] = JSON.parse(packageJsonStr); } return this.packageJsonCache[packageJsonPath]; } parseSpdxLicenseExpression(spdxLicenseExpression, context) { let parsedLicense; try { parsedLicense = spdxParse(spdxLicenseExpression); if (spdxLicenseExpression.indexOf('AND') !== -1) { this.logger.warn(`The SPDX license expression '${spdxLicenseExpression}' associated to ${context} ` + 'contains an AND operator, this is currently not properly handled and erroneous ' + 'licenses information may be provided to LibreJS'); } } catch (e) { this.logger.warn(`Unable to parse the SPDX license expression '${spdxLicenseExpression}' associated to ${context}.`); this.logger.warn('Some generated JavaScript assets may be blocked by LibreJS due to missing license information.'); parsedLicense = {'license': spdxLicenseExpression}; } return parsedLicense; } spdxToWebLabelsLicense(spdxLicenceId) { for (let i = 0; i < spdxLicensesMapping.length; ++i) { if (spdxLicensesMapping[i]['spdx_ids'].indexOf(spdxLicenceId) !== -1) { let licenseData = Object.assign({}, spdxLicensesMapping[i]); delete licenseData['spdx_ids']; delete licenseData['magnet_link']; licenseData['copy_url'] = ''; return licenseData; } } this.logger.warn(`Unable to associate the SPDX license identifier '${spdxLicenceId}' to a LibreJS supported license.`); this.logger.warn('Some generated JavaScript assets may be blocked by LibreJS due to missing license information.'); return { 'name': spdxLicenceId, 'url': '', 'copy_url': '' }; } spdxToWebLabelsLicenses(spdxLicenses) { // This method simply extracts all referenced licenses in the SPDX expression // regardless of their combinations. // TODO: Handle licenses combination properly once LibreJS has a spec for it. let ret = []; if (spdxLicenses.hasOwnProperty('license')) { ret.push(this.spdxToWebLabelsLicense(spdxLicenses['license'])); } else if (spdxLicenses.hasOwnProperty('left')) { if (spdxLicenses['left'].hasOwnProperty('license')) { let licenseData = this.spdxToWebLabelsLicense(spdxLicenses['left']['license']); ret.push(licenseData); } else { ret = ret.concat(this.spdxToWebLabelsLicenses(spdxLicenses['left'])); } ret = ret.concat(this.spdxToWebLabelsLicenses(spdxLicenses['right'])); } return ret; } extractLicenseInformation(packageJson) { let spdxLicenseExpression; if (packageJson.hasOwnProperty('license')) { spdxLicenseExpression = packageJson['license']; } else if (packageJson.hasOwnProperty('licenses')) { // for node packages using deprecated licenses property let licenses = packageJson['licenses']; if (Array.isArray(licenses)) { let l = []; licenses.forEach(license => { l.push(license['type']); }); spdxLicenseExpression = l.join(' OR '); } else { spdxLicenseExpression = licenses['type']; } } let parsedSpdxLicenses = this.parseSpdxLicenseExpression(spdxLicenseExpression, `module ${packageJson['name']}`); return this.spdxToWebLabelsLicenses(parsedSpdxLicenses); } copyFileToOutputPath(srcFilePath, ext = '') { - if (this.copiedFiles.has(srcFilePath) || srcFilePath.indexOf('://') !== -1) { + if (this.copiedFiles.has(srcFilePath) || srcFilePath.indexOf('://') !== -1 || + !fs.existsSync(srcFilePath)) { return; } let destPath = this.cleanupPath(srcFilePath); let destDir = path.join(this.weblabelsOutputDir, ...destPath.split('/').slice(0, -1)); this.recursiveMkdir(destDir); destPath = path.join(this.weblabelsOutputDir, destPath + ext); fs.copyFileSync(srcFilePath, destPath); this.copiedFiles.add(srcFilePath); } recursiveMkdir(destPath) { let destPathSplit = destPath.split('/'); for (let i = 1; i < destPathSplit.length; ++i) { let currentPath = path.join('/', ...destPathSplit.slice(0, i + 1)); if (!fs.existsSync(currentPath)) { fs.mkdirSync(currentPath); } } } }; module.exports = GenerateWebLabelsPlugin; diff --git a/swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/jslicenses.ejs b/swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/jslicenses.ejs index 329c7733..59ec6ce4 100644 --- a/swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/jslicenses.ejs +++ b/swh/web/assets/config/webpack-plugins/generate-weblabels-webpack-plugin/jslicenses.ejs @@ -1,81 +1,87 @@ <%# Copyright (C) 2019 The Software Heritage developers See the AUTHORS file at the top-level directory of this distribution License: GNU Affero General Public License version 3, or any later version See top-level LICENSE file for more information %> jslicense-labels1

Web Labels

<% for (let jsasset of Object.keys(jslicenses_data).sort((a, b) => { let va = a.split('/').slice(-1)[0]; let vb = b.split('/').slice(-1)[0]; if (va < vb) { return -1 } else if (va > vb) { return 1; } return 0; })) { let bundled_js_srcs = jslicenses_data[jsasset]; %> <% } %>
Script Licenses Sources
- <%= jsasset.split('/').slice(-1)[0] %> + + <% if (jsasset.split('/').slice(-1)[0]) { %> + <%= jsasset.split('/').slice(-1)[0] %> + <% } else { %> + <%= jsasset %> + <% } %> + <% for (let i = 0; i < bundled_js_srcs.length ; ++i) { let js_src = bundled_js_srcs[i]; for (let j = 0; j < js_src.licenses.length; ++j) { let js_license = js_src.licenses[j]; %> <%= js_license.name %> <% if (js_license.copy_url) { %> (view) <% } %> <% if (j != js_src.licenses.length - 1) {%>
<% } %> <% } %> <% if (i != bundled_js_srcs.length - 1) {%>

<% } %> <% } %>
<% for (let i = 0; i < bundled_js_srcs.length ; ++i) { let js_src = bundled_js_srcs[i]; %> <%= js_src.id %> <% for (let j = 0 ; j < js_src.licenses.length - 1; ++j) { %>
<% }%> <% if (i != bundled_js_srcs.length - 1) {%>

<% } %> <% } %>
\ No newline at end of file diff --git a/swh/web/templates/jslicenses.html b/swh/web/templates/jslicenses.html index cd83a645..40f114a6 100644 --- a/swh/web/templates/jslicenses.html +++ b/swh/web/templates/jslicenses.html @@ -1,73 +1,77 @@ {% extends "layout.html" %} {% comment %} Copyright (C) 2019 The Software Heritage developers See the AUTHORS file at the top-level directory of this distribution License: GNU Affero General Public License version 3, or any later version See top-level LICENSE file for more information {% endcomment %} {% load swh_templatetags %} {% block title %}JavaScript license information{% endblock %} {% block navbar-content %}

JavaScript license information

{% endblock %} {% block content %}

This page states the licenses of all the JavaScript files loaded by that web application. The loaded JavaScript files correspond to bundles concatenating multiple source files. You can find the details of the content of each bundle in the Web Labels table below.

{% for jsasset, bundled_js_srcs in jslicenses_data %} {% endfor %}
Script Licenses Sources
- {{ jsasset | split:"/" | last }} + {% if jsasset|split:"/"|last %} + {{ jsasset | split:"/" | last }} + {% else %} + {{ jsasset }} + {% endif %} {% for js_src in bundled_js_srcs %} {% for js_license in js_src.licenses %} {{ js_license.name }} {% if js_license.copy_url %} (view) {% endif %} {% if not forloop.last %}
{% endif %} {% endfor %} {% if not forloop.last %}

{% endif %} {% endfor %}
{% for js_src in bundled_js_srcs %} {{ js_src.id }} {% for js_license in js_src.licenses %} {% if not forloop.last %}
{% endif %} {% endfor %} {% if not forloop.last %}

{% endif %} {% endfor %}
{% endblock %}